Lattice QCD Production on Commodity Clusters at Fermilab

Large scale QCD Monte Carlo calculations have typically been performed on either commercial supercomputers or specially built massively parallel computers. Commodity clusters equipped with high performance networking equipment present an attractive alternative, achieving superior performance to price ratios and offering clear upgrade paths. The U.S. Department of Energy, through the SciDAC (Scientific Discovery through Advanced Computing) program, is supporting the investigation of commodity clusters as well as purpose built machines for lattice gauge computations. Lattice QCD codes are memory bandwidth, floating point, and network intensive. Successful design of clusters to run these codes requires knowledge of the bottlenecks which control performance. In this paper we examine a number of aspects of performance, observed while running the MILC lattice QCD code. For single systems, we discuss the effects of the various memory architectures. We examine optimizations such as data layout and SSE-assisted matrix algebra, and consider SMP behavior. We discuss scaling of the MILC code on Myrinet-connected clusters. Using a modified version of GM which allows control of bandwidth and latency, we examine the sensitivity of the code to network performance.